{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Filter by Tags in Zilliz Cloud Pipelines\n",
    "\n",
    "> (Note) Zilliz Cloud Pipelines is about to deprecate. Please stay tuned for detailed instructions on alternative solutions.\n",
    "\n",
    "In the previous [notebook](./zilliz_pipeline_rag.ipynb), we have learned the basics of Zilliz Cloud Pipelines. In this notebook, we show an example of filtering retrieval results by tags. The Pipelines operations are wrapped with a helper class to simply the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Setup\n",
    "### Prerequisites\n",
    "Please make sure you have a Serverless cluster in Zilliz Cloud. If not already, you can [sign up for free](https://cloud.zilliz.com/signup?utm_source=referral&utm_medium=partner&utm_campaign=2023-12-22_github-docs_pipeline-filter-notebook_github).\n",
    "\n",
    "To learn how to create a Serverless cluster and get your CLOUD_REGION, CLUSTER_ID, API_KEY and PROJECT_ID, please refer to this [page](https://docs.zilliz.com/docs/create-cluster) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "CLOUD_REGION = 'gcp-us-west1'\n",
    "CLUSTER_ID = 'your CLUSTER_ID'\n",
    "API_KEY = 'your API_KEY'\n",
    "PROJECT_ID = 'your PROJECT_ID'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Create an ingestion pipeline\n",
    "[Ingestion pipelines](https://docs.zilliz.com/docs/understanding-pipelines#ingestion-pipelines) can transform unstructured data into searchable vector embeddings and store them in Zilliz Cloud Vector Database.\n",
    "\n",
    "In the following example we create an Ingestion pipeline named as `my_ingestion_pipeline`. As part of creating the Ingestion pipeline, a vector database collection named `my_rag_collection` will be created in the cluster. It contains five fields:\n",
    "- `doc_name`, `chunk_id`, `chunk_text`, `embedding` as defined by `INDEX_DOC` function\n",
    "- `version` as defined by `PRESERVE` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pipeline_utils import IngestionPipeline\n",
    "\n",
    "collection_name = 'my_rag_collection'\n",
    "\n",
    "functions = [\n",
    "    {\n",
    "        \"name\": \"index_my_doc\",\n",
    "        \"action\": \"INDEX_DOC\",\n",
    "        \"inputField\": \"doc_url\",\n",
    "        \"language\": \"ENGLISH\",\n",
    "        \"chunkSize\": 500,\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"keep_doc_info\",\n",
    "        \"action\": \"PRESERVE\",\n",
    "        \"inputField\": \"version\",\n",
    "        \"outputField\": \"version\",\n",
    "        \"fieldType\": \"VarChar\"\n",
    "    }\n",
    "]\n",
    "\n",
    "ingestion_pipeline = IngestionPipeline(cloud_region=CLOUD_REGION,\n",
    "                                       cluster_id=CLUSTER_ID,\n",
    "                                       api_key=API_KEY,\n",
    "                                       project_id=PROJECT_ID,\n",
    "                                       collection_name=collection_name,\n",
    "                                       pipeline_name='my_ingestion_pipeline',\n",
    "                                       functions=functions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If you run this code and get the error \"This collection already exists\", it means you have created this collection before. You can change the `collection_name` or delete the collection manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Create a search pipeline\n",
    "[Search pipelines](https://docs.zilliz.com/docs/understanding-pipelines#search-pipelines) enables semantic search by converting a query string into a vector embedding and then retrieving top-K nearest neighbour vectors and doc chunks.\n",
    "\n",
    "In the following example we create a Search pipeline named `my_search_pipeline`. It searches the collection created by the Ingestion pipeline above.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pipeline_utils import SearchPipeline\n",
    "\n",
    "functions = [\n",
    "    {\n",
    "        \"name\": \"search_chunk_text\",\n",
    "        \"action\": \"SEARCH_DOC_CHUNK\",\n",
    "        \"inputField\": \"query_text\",\n",
    "        \"clusterId\": CLUSTER_ID,\n",
    "        \"collectionName\": collection_name\n",
    "    }\n",
    "]\n",
    "\n",
    "search_pipeline = SearchPipeline(\n",
    "    cloud_region=CLOUD_REGION,\n",
    "    api_key=API_KEY,\n",
    "    project_id=PROJECT_ID,\n",
    "    pipeline_name='my_search_pipeline',\n",
    "    functions=functions,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Run ingestion pipeline\n",
    "\n",
    "Ingestion pipeline accepts files from Object Storage Service such as [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html) or [Google Cloud Storage (GCS)](https://cloud.google.com/storage/docs/uploads-downloads).\n",
    "\n",
    "We use two versions of Milvus (an open-source vector database) doc, which are from [Milvus 2.3](https://milvus.io/docs/delete_data.md) and [Milvus 2.2](https://milvus.io/docs/v2.2.x/delete_data.md) . They are stored on Google Cloud Storage and attach its version info. We pass the version information into the keyword arguments of the `run()` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gcs_url_23 = 'https://publicdataset.zillizcloud.com/milvus_doc.md'  # The latest milvus 2.3 version documentation\n",
    "gcs_url_22 = 'https://publicdataset.zillizcloud.com/milvus_doc_22.md'  # Milvus 2.2 version documentation\n",
    "\n",
    "ingestion_pipeline.run(gcs_url=gcs_url_22, version='2.2')\n",
    "ingestion_pipeline.run(gcs_url=gcs_url_23, version='2.3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now we have successfully ingested the document by splitting it into doc chunks and uploading the generated embedding into the vector database collection. If you want to inspect the data in the collection, you can use the Data Preview tool in [Zilliz Cloud web UI](https://cloud.zilliz.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Build RAG application with Search pipeline\n",
    "\n",
    "### Run search pipeline\n",
    "We can use the `run()` method to run a search pipeline. The `run()` method takes a question as input and returns the top k knowledge fragments.\n",
    "The returned information also needs to include `other_output_fields=['version']`, and the filter condition is `version == \"2.2\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'chunk_text': '# Delete Entities\\nThis topic describes how to delete entities in Milvus.  \\nMilvus supports deleting entities by primary key filtered with boolean expression.  \\nDeleted entities can still be retrieved immediately after the deletion if the consistency level is set lower than Strong.\\nEntities deleted beyond the pre-specified span of time for Time Travel cannot be retrieved again.\\nFrequent deletion operations will impact the system performance.',\n",
       "  'version': '2.2'},\n",
       " {'chunk_text': '# Delete Entities\\n## Prepare boolean expression\\nPrepare the boolean expression that filters the entities to delete.  \\nMilvus only supports deleting entities with clearly specified primary keys, which can be achieved merely with the term expression in. Other operators can be used only in query or scalar filtering in vector search. See Boolean Expression Rules for more information.  \\nThe following example filters data with primary key values of 0 and 1.  \\n```python\\nexpr = \"book_id in [0,1]\"\\n```',\n",
       "  'version': '2.2'}]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = 'Can user delete milvus entities through non-primary key filtering?'\n",
    "search_pipeline.run(question=question, top_k=2, other_output_fields=['version'], filter='version == \"2.2\"', )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let’s try changing the filter conditions to `version == \"2.3\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'chunk_text': '# Delete Entities\\nThis topic describes how to delete entities in Milvus.  \\nMilvus supports deleting entities by primary key or complex boolean expressions. Deleting entities by primary key is much faster and lighter than deleting them by complex boolean expressions. This is because Milvus executes queries first when deleting data by complex boolean expressions.  \\nDeleted entities can still be retrieved immediately after the deletion if the consistency level is set lower than Strong.\\nEntities deleted beyond the pre-specified span of time for Time Travel cannot be retrieved again.\\nFrequent deletion operations will impact the system performance.  \\nBefore deleting entities by comlpex boolean expressions, make sure the collection has been loaded.\\nDeleting entities by complex boolean expressions is not an atomic operation. Therefore, if it fails halfway through, some data may still be deleted.\\nDeleting entities by complex boolean expressions is supported only when the consistency is set to Bounded. For details, see Consistency.',\n",
       "  'version': '2.3'},\n",
       " {'chunk_text': '# Delete Entities\\n## Prepare boolean expression\\nPrepare the boolean expression that filters the entities to delete.  \\nMilvus supports deleting entities by primary key or complex boolean expressions. For more information on expression rules and supported operators, see Boolean Expression Rules.',\n",
       "  'version': '2.3'}]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_pipeline.run(question=question, top_k=2, other_output_fields=['version'], filter='version == \"2.3\"', )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We can see that when we ask a question, this search run can return the top k knowledge fragments we need. This is also a basis for forming RAG."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Build a chatbot powered by RAG \n",
    "Below, we show a simple RAG app that can answer based on the knowledge we have ingested previously. It uses OpenAI `gpt-3.5-turbo` as LLM and a simple prompt. To test it, you can replace with your own OpenAI API Key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI()\n",
    "client.api_key = os.getenv('OPENAI_API_KEY')  # your OpenAI API key\n",
    "\n",
    "\n",
    "class MilvusDocChatbot:\n",
    "    def __init__(self, search_pipeline):\n",
    "        self.search_pipeline = search_pipeline\n",
    "\n",
    "    def retrieve(self, query: str, milvus_version: str) -> list:\n",
    "        \"\"\"\n",
    "        Retrieve relevant text with Zilliz Cloud Pipelines.\n",
    "        \"\"\"\n",
    "        results = self.search_pipeline.run(question=query, top_k=2, other_output_fields=['version'], filter=f'version == \"{milvus_version}\"', )\n",
    "        return results\n",
    "\n",
    "    def generate_answer(self, query: str, context_str: list) -> str:\n",
    "        \"\"\"\n",
    "        Generate answer based on context, which is from the result of Search pipeline run.\n",
    "        \"\"\"\n",
    "        completion = client.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\",\n",
    "            temperature=0,\n",
    "            messages=\n",
    "            [\n",
    "                {\"role\": \"user\",\n",
    "                 \"content\":\n",
    "                     f\"We have provided context information below. \\n\"\n",
    "                     f\"---------------------\\n\"\n",
    "                     f\"{context_str}\"\n",
    "                     f\"\\n---------------------\\n\"\n",
    "                     f\"Given this information, please answer the question: {query}\"\n",
    "                 }\n",
    "            ]\n",
    "        ).choices[0].message.content\n",
    "        return completion\n",
    "\n",
    "    def chat_with_rag(self, query: str, milvus_version: str) -> str:\n",
    "        context_str = self.retrieve(query, milvus_version=milvus_version)\n",
    "        completion = self.generate_answer(query, context_str)\n",
    "        return completion\n",
    "\n",
    "\n",
    "\n",
    "chatbot = MilvusDocChatbot(search_pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "This implements an RAG chatbot, it will use Search pipeline to retrieve the most relevant chunks from ingested documents, and enhance the answer quality with it. Let's see how it works in action!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Chat with RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'No, according to the provided information, Milvus only supports deleting entities by primary key filtered with a boolean expression. Other operators can be used only in query or scalar filtering in vector search.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = 'Can user delete milvus entities through non-primary key filtering?'\n",
    "chatbot.chat_with_rag(question, milvus_version='2.2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The ground truth content in the original knowledge text is:\n",
    "> **Milvus supports deleting entities by primary key filtered with boolean expression.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Yes, users can delete Milvus entities through non-primary key filtering by using complex boolean expressions.'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbot.chat_with_rag(question, milvus_version='2.3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The ground truth content in the original knowledge text is:\n",
    "> **Milvus supports deleting entities by primary key or complex boolean expressions**. Deleting entities by primary key is much faster and lighter than deleting them by complex boolean expressions. This is because Milvus executes queries first when deleting data by complex boolean expressions.\n",
    "\n",
    "\n",
    "Indeed, Milvus 2.3 has enhanced the [Delete Entities](https://milvus.io/docs/v2.2.x/delete_data.md) function. In the latest version 2.3, deleting by complex boolean expressions can be supported. By filtering different Milvus versions, we have achieved the ability to RAG different knowledge sources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's how to use Zilliz Cloud Pipelines to build RAG applications. To learn more, you can refer to https://docs.zilliz.com/docs/pipelines for detailed information.\n",
    "\n",
    "If you have any question, feel free to contact us at support@zilliz.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
